No-Code Artificial Intelligence by Agrawal Ambuj;

Author:Agrawal, Ambuj; [Agrawal, Ambuj]
Language: eng
Format: epub
Publisher: BPB Publications
Published: 2023-04-15T00:00:00+00:00

Figure 5.14: SageMaker Canvas building the AI model

In this section we looked at the model training steps to train and build the machine learning model. In the next section we will look at ways to evaluate our machine learning model based on how well it performed on our training dataset.

Evaluation

Machine Learning models are evaluated on different metrics based on the type of model used. The forecasting model is evaluated based on the situations for which they would be used. Generally the evaluation of forecasting models consists of four steps such as testing assumptions, testing the data quality, replicating outputs and assessing outputs.

With Amazon SageMaker Canvas, you can evaluate the model using the column impact and prediction accuracy. Prediction accuracy is calculated using the Weighted Average Percentage Error (WAPE) which is used to measure the accuracy of statistical forecasts compared to the actual or real outcomes for a sample. WAPE uses the sum of predicted values vs the sum of the observed value to compute the error. For our dataset, the prediction accuracy or WAPE is 76.184% (it might be slightly different when you train the model on your side due to randomness involved in generating the model).

Column impact tells you the percentage score of how much weight (or importance) a column has in making predictions compared to other columns. For our dataset, promo has 100% impact on the sales performance and school holiday has no impact on sales performance as shown in Figure 5.15:

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.